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CLAIMS : 



1 Data processing apparatus comprising: 

receiving means for receiving a data structure 
defining gra^r rules for recognition of a natural 

language input? 

analysing means for analysing the data structure to 
identify positions in the grammar rules at which pauses 
can occur in the natural language input; and 

generating means for generating a modified data 
structure defining modified grammar rules for recognition 
of a natural language input with pauses therein. 

2 Data processing apparatus according to claim 1, 
wherein said analysing means is adapted to identify the 
positions in accordance with pause criteria for the 
natural language input. 

3. Data processing apparatus according to claim 1 or 
claim 2 wherein said generating means is adapted to add 
^rKer means to the identified positions in the grammar 
ru les at which pauses can occur in the natural language 
input to generate the modified data structure. 
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4 • Data processing apparatus according to^afi^t^^^e^ag 
e-la^irr wherein said generating means is adapted to 
fragment the grammar rules in accordance with said 
identified positions to generate sub grammar rules to 
form said modified data structure. 

5/ Data processing apparatus according to claim 4, 
wherein said generating means is adapted to form a 
hierarchical structure using said sub grammar rules to 
form said modified data structure* 

Claim / or 2. 

6 . Data processing apparatus according te tany pr eced±af 
cl-aiffi-wherein said receiving means is adapted to receive 
a data structure defining grammar rules for use in speech 
recognition of a natural language speech input, and said 
generating means is adapted to generate said modified 
data structure defining modified grammar rules for speech 
recognition of a natural language speech input with 
pauses therein. 

Clam i or^ 

7* Data processing apparatus according to^ any preoedirra? 
oJ.ain » wherein said receiving means is adapted to receive 
said data structure defining grammar rules for 
recognition of a natural language input as a first 
modality input in conjunction with associated events in 



at least, one further modality input r said data structure 
defining the association between events in each modality 
input, and events in said first modality input comprising 
units in the natural language. 

8. Data processing apparatus according to claim 7 w 
wherein said analysing means is adapted to identify said 
positions in the grammar rules based on events in at 
least one said further modality input, 

9. Data processing apparatus comprising; 
receiving means for receiving a data structure 

defining grammar rules for recognition of a natural 
language input; 

analysing means for analysing the data structure to 
identify positions in the grammar rules at which events 
in at least one further modality input can occur in the 
natural language input as a first modality input; and 

generating means for generating a modified data 
structure defining modified grammar rules for recognition 
of a natural language input as the first modality input 
in conjunction with associated events in a said further 
modality input r said data structure defining the 
association between events in each modality input , and 



events in said first modality input comprising units in 
the natural language. 

10. Data processing apparatus according to claim 9 r 
wherein said analysing means is adapted to identify 
positions in the grammar rules at which pauses can occur 
in the natural language input based on events in at least 
one said further modality input. 

11- Data processing apparatus according to claim — 8-, 
9 or 10 , wherein said generating means is adapted to 
generate a further modified data structure defining said 
modified grammar rules and the relationships with events 
in the or each further modality input. 

12. A data processing method comprising the steps: 

receiving a data structure defining grammar rules 

for recognition of a natural language input; 

analysing the data structure to identify positions 

in the grammar rules at which pauses can occur in the 

natural language input; and 

generating a modified data structure defining 

modified grammar rules for recognition of a natural 

language input with pauses therein. 



13. A data processing method according to claim 12, 
wherein the analysing step includes identifying the 
positions in accordance with pause criteria for the 
natural language input. 

14 - A data processing method according to claim 12 or 
claim 13 r wherein the generating step includes adding 
marker means to said data structure to identify the 
positions in the grammar rules at which pauses can occur 
in the natural language input. 

15, A data processing method according to a-xvy — ©r*e — ct£ 

Clainrtf or \2> 

^■s laiiR s — 2r2 — to — ±=<£ wherein the generating step includes 
fragmenting the grammar rules in accordance with said 
identified positions to generate sub grammar rules to 
form said modified data structure. 

16. A data processing method according to claim 15 
wherein the generating step includes forming a 
hierarchical structure using said sub grammar rules to 
form said modified data structure. 

17- A data processing method according to af*y — one — 

ClCXu/i \*a Or iS 

/y©ia^fem«=i~2 — to 16 , wherein the receiving step comprises 
receiving a data structure defining grammar rules for use 



in speech recognition of a natural language speech input , 
and the generating step comprises generating a modified 
data structure defining modified grammar rules for speech 
recognition of a natural language speech input with 
pauses therein. 



18 . A data processing method according to any — one — ef- 
efa-im — 1-2 — to — *3r7-, wherein the receiving step comprises 
receiving a data structure defining grammar rules for 
recognition of a natural language input as a first 
modality input in conjunction with associated events in 
at least one further modality input, said data structure 
defining the association between events in each modality 
input r events in said first modality input comprising 
units in the natural language, 

19. A data processing method according to claim 18, 
wherein the analysing step comprises identifying said 
positions in the grammar rules based on events in at 
least one said further modality input . 

20. A data processing method comprising! 

receiving a data structure defining grammar rules 
for recognition of a natural language input; 
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analysing -the data structure to identify positions 
in the grammar rules at which events in at least one 
further modality input can occur in the natural language 
input as a first modality input; and 

generating a modified data structure defining 
modified grammar rules for recognition of a natural 
language input as the first modality input in conjunction 
with associated events in a said further modality input, 
said data structure defining the association between 
events in each modality input, and events in said first 
modality input comprising* units in the natural language - 

21. A data processing method according to claim 20, 
wherein the analysing step comprises identifying 
positions in the grammar rules at which pauses can occur 
in the natural language input based on events in at least 
one said further modality input, 

22- A data processing method according to claim 3*7 — ±9^ 
-S M 3 or- 21, wherein the generating step includes generating 
a further modified data structure defining said modified 
grammar rules and the relationships with events in the or 
each further modality input. 



23. Apparatus for generating data in a computer usable 
form, the apparatus comprising s 

receiving means for receiving a natural language 
input with a number of pauses therein; and 

recognition means for recognising said natural 
language input using the modified data structure 

claims \^\%xoov^\ 

generated using the method of any one o f ^G-laims-— 12 to 22 
to generate data in computer usable form. 

24- Apparatus according to claim 23 , wherein said 
recognising means comprises speech recognition means for 
recognising a natural language speech input. 

25- A method of generating data in a computer usable 
form, the method comprising receiving a natural language 
input with a number of pauses therein; and 

recognising said natural language input using the 
modified data structure generated using the method of «ay 
^^2^J^iaims — 3r2 — to— %2r to generate data in computer 
usable form. 

26- A method according to claim 25, wherein the 
recognising step comprises speech recognition of a 
natural language speech input. 
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27- Apparatus for genera-ting data in a computer usable 
form, the apparatus comprising: 

first modality receiving data generated for a 
natural language input by the apparatus of claim 23 -o^ 
ciaim^^l , said data comprising recognised units of the 
natural language and comprising data of a first modality 
input ; 

further modality receiving means for receiving data 
identifying events in at least one further modality 
input; 

data structure receiving means for receiving a 
further modified data structure defining modified grammar 
rules and the relationships with events in the or each 
further modality, said further modified data structure 
having been generated using the method of claim 18; 

analysing means for analysing the first modality 
input data and the or each further modality input data to 
determine if they match with any said modified grammar 
rule and related events in the or each further modality; 
and 

generating means for generating computer usable data 
in dependence upon said analysis by said analysing means. 

28- Apparatus according to claim 27, wherein said first 
modality receiving means is adapted to receive 
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recognition data comprising an ordered list of likely- 
natural language units to accompany the most likely 
natural language for each natural language unit 
recognised, and said analysing means is adapted to use 
said ordered list vhen the most likely natural language 
units do not result in a match with any modified grammar 
rule and related events in the or each further modality - 

29, Apparatus according to claim 27 o^e— claim 2S - wherein 
said first modality receiving means is adapted to receive 
speech recognition data. v 

30. A method of generating data in a computer usable 
form / the method comprising: 

a first receiving step of receiving data generated 
for a natural language input by the method of claim 25 
glQim-^ v said data comprising recognised units of the 
natural language, and comprising data of a first modality 
input ; 

a second receiving step of receiving data 
identifying events in at least one further modality 
input } 

a third receiving step of receiving a further 
modified data structure defining modified grammar rules 
and the relationship with events in the or each further 



modality, said further modified data structure having 
been generated during the method of claim 22; 

analysing the first modality input data and the or 
each further modality input data to determine if they 
match with any said modified grammar rule and related 
events in the or each further modality; and 

generating computer usable data in dependence upon 
said analysis. 

31- A method according to claim 30 wherein the first 
receiving step comprises receiving recognition data 
comprising an ordered list of likely natural language 
units to accompany the most likely natural language unit 
for each natural language unit recognised, and the 
analysis step includes using the ordered list when the 
most likely natural language units do not result in a 
match with any modified grammar rule and related events 
in the or each further modality - 

32. A method according to^ra™=^-or claim 31, wherein 
the first receiving step receives speech recognition 
data . 



4 



60 

33. Processor implementable instructions for controlling 
a processor to carry out the method of any one of claims 
A y^l3-W==22 , 25, 2-6— oi- 30 - t o 32 . 

5 34- A carrier medium for carrying the processor 

implement able instructions according to claim 33. 



